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Preface 



As education reformers have blazed new trails throughout the decade, they have increasingly 
sought better maps of where they have been and where they are going. The discussion of this 
mapping has coalesced around the term "education indicator systems." While considerable effort 
has gone into defining and producing such systems by groups such as the Council of Chief State 
School Officers, the National Center for Education Statistics, the National Research Council, the 
National Science Foundation, the RAND Corporation, and individual states, all would agree that 
there is a long way to go, both technically and politically. 

The challenge was stated by Marshall Smith in Phi Delta Kapp an: 

Irdicators are designed to reflect the health and efficiency 

of an education system and to point the way toward its improvement. 

To develop a set of indicators that does these things 
requires an image of a fully functioning system. 

In State ^iducation Indicators: Measur ed Strides. Missing Steps , Stephen S. Kaagan 
and Richard J. Coley offer their image of an education indicator system, with optimism and 
clarity of purpose. This publication results from a project jointly funded by the Center for Folic;,' 
Research in Education at Rutgers University and Educational Testing Service. It combines the ex- 
perience of being a chief state school officer during the reform period (Kaagan), formal and infor- 
mal visits to a number of states (both authors), and familiarity with the recent work of others, to 
arrive at some views about "right vision" for the future. It is addressed more to practitioners than 
to academics, and passes on some lessons that have emerged in the creation and operation of in- 
dicator systems. 

We are pleased to make these views available to the education community. 

Susan Fuhrman 
Director 

Center for Policy Research 

in Education 
Rutgers University 

Paul E. Barton 
Director 

Policy Information Center 
Educational Testing Service 
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Introduction 



As educational reforms have been implemented in the 1980s, 
attention has turned to the questions: What differences arc 
they making, and how will we know? According to the Nation- 
al Conference of State Legislatures, "Accountability is hot"; it 
was identified as a top priority for 1989 by committee chairs in 
31 states ( Education Week . January 18, 1989). This report is 
about one important means of achieving accountability: the in- 
troduction of statewide ''education indicator systems/' We aim 
to describe the state of the art in the develop-nent of such sys- 
tems, identify critical points in the path of their development 
and improvement, and point out ways of judging whether these 
systems are succeeding or failing. The report reflects the 
belief that measurement of key aspects of a state's education 
system, such as the performance of students and the various 
factors affecting that performance, will continue to be a 
**growth industry" as state leadership applies its hand to the 
challenge of improving schools. 

After an inspiring level of state educational reform ac- 
tivity during the last five years, policy makers are looking for 
data to provide clues to further policy development and to fine- 
tune existing reform programs. National reports like A N ation 
at Risk , as well as efforts by the Secretary of Educatio i to com- 
pare states on the basis of education indicators, pointed oi't the 
meager quality and quantity of education statistics in our 
country, particularly at the state level. Public officials and 
citizens alike want to know what their investments in public 
education have bought. State education agencies are respond- 
ing by building new information systems and seeking ways of 
applying them. 

This report is divided into three sections. The first sec- 
tion describes the genesis of education indicator systems and 
discusses important definitions and necessary components. 
The focus of the section is on the central features of such sys- 
tems and on the issues that must be addressed with regard to 
their purposes, applications, and effects at both state and local 
levels. T||e second section provides vignettes of state educa- 
tion indicator systems in the four states we visited. Tiiese 
descriptions make the conclusions and recommendations that 
follow more comprehensible by providing appropriate context 



This report is about one 
important means of 
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and substantiation. Ex' .iples of state practices point to 
areas of concern that must be given attention as indicator 
systems are developed. The third section presents the 
study's conclusions and recommendations, focusing on 
those that reflect multistate or national tendencies. The sec- 
tion closes with some necessary steps that the authors 
believe must be taken if state indicator systems are to be- 
come useful policy tools at both state and local levels. 

A word about how we prepa'-ed this report. One of 
us (Kaagan) served as a Chief State School Officer (Ver- 
mont) for six years and was heavily involved with the Coun- 
cil of Chief State School Officers in its new effort to 
support the development of a state-by-state comparison sys- 
tem. A review of the growing body of literature on educa- 
tion indicator systems was followed by a field study with 
the following elements: 

• Formal site visits in four states that are considered by 
many to be front-runners in the development and use of 
indicator systems (California, Connecticut, New York, 
and South Carolina). A structured interview protocol 
guided discussions with state education agency staff, 
governor's office staff, legislators and legislative staff, 
state business and community leaders, education associa- 
tion representatives, and selected local school officials. 

• Analysis of data and interview results from a six-state 
database developed by the Center for Policy Research in 
Education to study the effects of state-level reforms of 
the 1980s (Arizona, Califorria, Florida, Georgia, Min- 
nesota, and Pennsylvania). 

• Substantial involvement with Missouri education 
leaders during 1988 and 1989 to assist them develop a 
state education indicator system. 

• Informal visits to four states (Arkansas, Minnesota, 
Texas, and Washington). 

Several major academic studies on education in- 
dicators that we believe provide a sound conceptual base 
have recently been published. What we report here emerges 
from our perceptions of state experience in the development 
and use of state indicator systems; we believe some lessons 
are accumulating that v.ill benefit the growing number of 
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people using indicator systems. A useful summary of key char- 
acteristics of state indicator systems is provided in Figure 1, 
"Key Dimensions of the 50-State Performance Accountability 
Systems" (OERI, 1988). 

What Indicator 
Systems Are 
and Why They 
Are Important 



The Changing State Role 



State leaders have always been involved with educa- 
tion management. In the last five years, they have entered the 
arena of school management, and it is unlikely that they will 
withdraw in the years ahead. What this means is that gover- 
nors, legislators, and state education agency leaders are no 
longer operating on the outer edges of the school business, 
modulating school finance mechanisms and making school 
redistricting arrangements. Instead, they are smack in the mid- 
dle of the action, wrestling with matters of school effective- 
ness, teacher policy, curriculum emphases, and student 
performance. And in many state education agencies, the focus 
of attention and activity has changed from proces:; (e.g., num- 
ber of hours per week of math) to substance (e.g., the content 
of the math courses). This has happened whether the state is 
more accustomed to centralized management of schools or is 
steeped in the tradition of local control. States have decided in 
the last half decade to be major actors on the core issues of 
schooling and have looked for policy instruments to affect 
those core issues. States with more centralized approaches to 
school governance have tended to rely more on mandates and 
to be more directive, such as when they pass laws specifying 
statewide curriculum. Those more decentralized have tended 
to rely less on mandates and more on incentives that allow sub- 
stantial local latitude, such as project grants that allow 
localities choices in their use. 



Governors, legislators, 
and state education agen- 
cy leaders are no longer 
operating on the outer 
edges of the school busi- 
ness... Instead, they are 
smack in the middle of 
the action, wrestling with 
matters of school effec- 
tiveness, teacher policy, 
curriculum emphases, 
and student performance. 



The net result is that the state has asserted itself as a top 
manager of school operations . This does not mean, except in 
rare instances, that the state has become the line manager of 
schools, but that it has endeavored to take more control, 
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Figure 1 

KEY DIMENSIONS OF THE 50 STATE PERFORMANCE ACCOUNTABILITY SYSTEMS 
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Figure 1 ( continued ) 
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SOURCE: Council of Chief SUtc School Offlccra 1987 Survey and related State documenU. 

1 The District of Columbia and Hawaii each operate a single system In which the State and the district are the same. 

2 Missouri and North Dakota send schooMplus North Dakota district)— level data to parents but not to the press. 

3 Montana and North Dakota offer local districts the option of using a State achievement test. 



Reprinted from: Creating Responsible and Responsive Accountability Systems. 
U.S. Department of Education. Office of Educational Research and Improvement. 
September 1988. pp. 28-29 
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whether subtly through capacity-building measures or not so 
subtly through mandates. 

The Place of Education Indicator Systems 

Principally, this state control is being expressed 
through the monitoring of local district compliance with new 
or expanded bodies of rales or regulations, through a variety 
of measures intended to upgrade the quality of school staff, 
and strategic allocations of material resources and recogni- 
tion (e.g., monetary "Awards, project grants, honors for excel- 
lence) to encourage desired behavior. Another consequen- 
tial way the state presence is increasingly felt is through the 
development of state indicator systems — selected forays 
into local opei Jtions to glean information about what is hap- 
pening and why and to identify problem areas that should be 
addressed. These systems often entail the development of 
new or expanded data systems. 

Indicators and Indicator Systems Defined 

Caretully crafted and gathered statistics are the basic 
building blocks of indicator systems. 7! ^se sta isfics are 
derived from a test, a survey, or collection of information on 
important aspects of the education system. Ideally, they are 
derived from tests, surveys, or data collection carried out pe- 
riodically to produce comparable data over time. The statis- 
tics are designed to be useful in describing some quantitative 
or qualitative aspect of the education system. In creating a 
system useful for policy, judgment c^^h arc necessary in 
selecting "indicators" or statistical series, for some measures 
have greater currency, or present value, than others. To il- 
lustrate, the number of high school age students who drop 
out of school is a more valued statistic than the total avail- 
able square footage of a system's schools; there is a strong 
constituency concerned about dropouts, with enough data 
available to document a problem. It would not be hard, how- 
eve^ to imagine the latter measure becoming important if 
available classroom space was halved in the next few years. 
In coi.trast to designs for evaluation of specific changes, in- 
dicator systems need to have open architecture and con- 
tinuity over a period of time to be useful when an aspect of 
the educational system becomes an issue . 
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Accordingly, the challenges in developing an education 
indicator system are to get key constituencies involved in ar- 
ticulating the goals of education, and then to quantify these fac- 
tors and gather the necessary data. This is not an easy task. 
Getting consensus on what is iniportant, quantifying it, and ob- 
taining the data are formidable, but necessary, steps in the 
development of an indicator system. To summarize, an in- 
dicator is a statistic that is judged important; it is a "vital sign ' 
regarding the "health" of the t tiucational program. An in- 
dicator system is a comprehensive set of such vital signs. (For 
a good, comprehensive description of education indicator sys- 
tems, seeOakes, 1988.) 

Looked at as a number on a bairen page, a statistic, how- 
ever important the phenomenon it measures, has modest useful- 
ness. Only when its value falls short of or exceeds a level of 
legitimate comparison does it become significantly useful. A 
dropout rate of 10 percent for a high school may appear accept- 
able until it is shown to be twice that of a similar school down 
the road, or not to meet the community's need for full participa- 
tion in school. The usefubiess of an indicator rests on its ability 
to show what happens over time, what it can say about the per- 
formance of a school or district compared to other schools or 
districts, or how the condition it measures compares with socie- 
tal reeds or expectations. 

An indicator system is a framework into which an array 
of mdicators are placed for review and analysis, leading to 
necessary modifications of policy and practice. Indicators are 
placed so that relationships among them can be examined; the 
framewoiic that has gained general acceptance for presenting 
data in such a way is the "inpi i, process, outcome" model 
shown in Figure 2 (Shavelson, 1987). This model demonstrates 
how input, process, and outcome components can be reuted to 
each other. 



The usefulness of an 
indicator rests on iis 
ability to show what 
happens over time, 
what it can say 
about the perfor- 
mance of a school or 
district compared to 
other schools or dis' 
tricts, or how the 
condition it 
measures compares 
with societal needs 
or expectations. 



In this model, "outcome" comprises some results ex- 
pected of schooling; "process" identifies the school ingredients 
most directly responsible for outcomes; and "input" represents 
certain extremely important, but less malleable, school and non- 
school characteristics that have well documented effects on out- 
comes. These input or context variables are seldom witliin the 
control of the school, particularly in the short term. An ex- 
ample of ui outcome measure is student knowledge as assessed 
by an achievement test; process, the amount of time an elemen- 
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Pifure 2: Linking Elements of the Educational System 
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Sourc*: Richard Shavelson, et al. Indicator Systems Jor Monitoring Mathematics 
and Science Mucation. the RAND Corporation, August 1987 



tary teacher dedicates per week to science instruction; and 
input, the percentage of children from low-income households. 



For an indicator system to reach its full potential, the 
willingness and capability to encompass inputs, processes , and 
outcomes and establish the relationships among them, are essen- 
tial. The important information provided by performance statis- 
tics is enriched when viewed in relationship to the prime 
components of the system. The use of any statistic for policy 
purposes carries an implicit model of the dynamics of a system; 
it is better to make the model explicit and increase under- 
standing of the statistic's significance. Leadership — at all 
levels — has to wrestle with these relationships i^ indicator sys- 
tems are to inform policy and if the benefits are to match or ex- 
ceed the costs of development. Put simply, the reporting of test 
scores alone does not make an indicator system; it may show 
ups and downs but will not lead to any understanding of why 
changes are occurring. Construing relationships among in- 
dicators is not an automatic exercise; judgment remains essen- 
tial. But ordering and examining relationships (as in the 
diagram) are critical if the indicator system is to reveal what 
leads to success or failure. 

The Prime Purpose of State Indicator Systems 

With some knowledge of what an indicator and an in- 
dicator system are, one can now turn to the more mtiiguing mat- 
ters of purposes and applications. For what purpose might in- 
dicator systems be deployed? For what purposes are they being 
deployec*.? Answers to these two questions will provide us with 
a clearer senre of the importance of indicator systems to tiie 
governance and management of education and a greater recogni- 
tion of the distance that needs to be traveled for the systems to 
reach their true potential. 

The essential purpose of education indicator systems is 
to assess direction, mission, and strategy. For state officials, 
this means reviewing and analyzing the aims of the state's 
education system and determining whether they are being met. 
Assessing aims is increasingly necessary as state leaders have 
become more outspoken about what schooling should achieve 
and have acted to impress their views on localities. 

Above all else then, indicator systems should assist state 
officials in determining 'Yight vision," a term used by Peter 
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An indicator system, 
in effect, is a device 
for setting organiza- 
tional direction, by 
reconciling aims 
with actions and 
making adjustments 
in response to effects 
and to revealed 
relationships among 
effects, school treat- 
ments, andcontex' 
tual variables. 



Drucker. TTie yield of the systems should provoke gover- 
nors, legislators, state boards of education, and state educa- 
tion agency leaders and staff to assay what the system is ac- 
complishing against what it should be accomplishing and 
ultimately to adjust their sights, i.e., set policy and regear 
programs appropriately. An indicator system, in effect, is a 
device for setting organizational direction, by reconciling 
aims with actions and making adjustments in response to ef- 
fects and to revealed relationships among effects, school 
treatments, and contextual variables. 

"Right vision" means that the individual creating the 
measure (indicator system) is responsible for making that 
measure congruent with the policy aims being pursued. 
"Right vision" bespeaks the state's top management respon- 
sibility for setting direction and assessing whether it is being 
pursued. It implies that the state, while designing an ac- 
countability system, be accountable for making the critical 
link between what is assessed and what schools are supposed 
to be doing. 

Drucker (1974a makes this point very well in 
delineating top management's measurement responsibilities: 

''That we can quantify something is no 
reason for measuring it. The question is *ls 
this what a manager should consider impor- 
tant?' 'Is this what a manager s atteniion 
should he focused on?' 'Is this a true state- 
ment of the basic realities of the enter- 
prise?' 'Is this the proper focus for control, 
that is, for effective direction with maximum 
economy of effort?' " 



and he adds later: 



"Because controls have such an impact, it 
is not only important that we select the right 
ones. To enable controls to give right 
vi.sion and to become the ground for effec- 
tive action, the measurement must also he 
appropriate. That is, it must present the 
events measured in structurally true form.'' 



Related Applications 



Applicaiioiis of indicator systems can be placed along a 
continuum from providing simple information at one end to 
labeling schools or districts as academically bankrupt at the 
other. An elementary application may reveal the effectiveness 
of the system to interested parties, presenting state and local 
government officials, societal organizations, parents, and educa- 
tion professionals with the data on outputs, processes, and in- 
puts. This provides grist for democratic decision making — 
letting citizens and their representatives know how well or poor- 
ly the schools are doing and providing some contextual informa- 
tion in which to couch the results. A second and more complex 
application is determining whether policies and programs are 
being successful, if what is being done in schools is making any 
difference, and if so, to whom and in what ways, A third is to 
suggest areas for further study that may produce evidence on 
which to base policy changes and pro^^ram shifts. 

An application of another sort, with a different purpose 
altogether, is to hold constituent entities and individuals ac- 
countable for results — local districts, including, perhaps, those 
hired to manage local school systems and those who provide 
educational services directly, administrators and teachers, 
The difference between what is termed above the essential pur- 
pose of indicator systems and this purpose is quite substantial. 
This purpose takes an indicator system well beyond providing 
data important to enforcing policy decisions and objectives, 
When there are direct coiisequences, tangible or intangible, at- 
tached to numbers, the burden on systems quality mushrooms. 
This function, of holding local operations accountable for per- 
formance, is markedly different from the use of the systems 
by state leaders to hold themselves accountable for determining 
policy direction, supporting it, and carrying it out. Both func- 
tions are equally important. 

Some are strong advocates of only the latter. A par- 
ticularly emphatic statement of this position is that of Peter 
Drucker: 



Applications of in- 
dicator systems can 
be placed along a 
continuum from 
providing simple in- 
formation at one end 
to labeling schools or 
districts as academic 
cally bankrupt at the 
other. 
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Measuring requires, first and foremost, 
analytical ability. But it also demands that 
measurement he used to make self-control 
possible rather than abused to control 
people from the outside and above — that 
is, to dominate them. It is the common 
abuse of this principle that largely explains 
why measurement is the weakest area in the 
work of the manager today '(Drucker. 
1974). 

Messages Indicators Send 

Finally, regardless of purpose, indicator systems send 
messages or signals, usually in overbearing and truncated 
tones, across the board lo all hearers (see David, 1988). By 
signalling what is important, they offer a unique opportunity 
for state policy makers to affect local education practice in a 
most efficient way. It is therefore essential that we under- 
stand what messages are intended, which ones are received, 
and by whom. The choice of signals to be sent should be 
made carefully; it should reflect state vision or goals and be 
sensitive to negative effects. 

We need to understand that fx)werful forces for stand- 
ardization are inherent in indicator systems, since they 
employ uniform metrics to assay what is happening in the 
real world. Olivia Golden, Lecturer at Harvard's Kennedy 
School of Government, has studied the tension between the 
need for standardization and accountability in large scale sys- 
tems versus the need for local discretion. What she says 
about universal standards and regulations applies with equal 
force to the yield of education indicator systems: 

'7// a large program where subtle, in- 
dividual interventions to improve quality 
are hard to carry out, politicians and 
managers will be tempted to impose univer- 
sal standards and regula:inns instead, par- 
ticularly given the concern for consistency. 
In a human services program where goals 
may be hard to specify, these regulations 
may have quite uncertain relations to the 
'*rear program quality that matters to 
results'' (Golden, 1988). 



Some State 
Developments 



This section contains brief vignettes of indicator systems 
in the four states visited for this study. These descriptions, sup- 
plemented from experience in other states, provide contexts, 
examples, and substantiation for the recommendations in the 
last section. 



Connecticut 



The momentum of education reform in Connecticut 
built up over a decade or so, spurred on by school finance 
reform legislation during the mid 1970s and a major education 
reform bill during the mid 1980s. Along with a steep increase 
in the state share of education funding came an increasing 
demand for accountability, both from the legislature and the 
business community. Connecticut's indicator system reflects 
the necessary political balance between this need for account- 
ability and a strong tradition of local school district control. 

Evaluation, research, and development are prominent 
functions of the state education agency. Agency officials are 
enterprising and have created an audience for their products 
among the media and the public. The Division of Research, 
Evaluation, and Assessment engages in abroad range of ac- 
tivities including criterion-referenced testing of basic and 
higher order skills, special research studies, special program 
evaluations, and teacher assessment activities. The department 
is also well connected to prominent national research and 
development efforts. 

The basis of Connecticut's accountability system is the 
State Board of Education's Design for Excellence: 
Connecticut's Comprehensive Plan for Elementary. Secondary. 
VocationaK C areer and Adult Education. 1986-1990 This 
report includes the state's goals. For each goal, objectives have 
been established and a set of education indicators have been 
selected to report progress in meeting them. Figure 3 provides 
an example of Connecticut's state-level indicator reporting. It 
shows state progress in meeting one of the State Board of 
Education's goals — to ensure equity for all children. Local 
districts are required to develop local goals and objectives that 



Connecticut's in' 
dicator system reflects 
the necessary political 
balance between this 
need for accountability 
and a strong tradition 
of local school district 
control. 
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Plgnre 3: Connecticut's State 
Level Indicator Reporting 




Indicators 
of Success 



f 

i 



Change is in the desired direction. 

Change is counter to the desired 
direction. 



There is little change 

Baseline data. No comparable data were 
available before 1985-86 

(5) No data indicating change are available. 



Goal I: To Ensure Equity for All Children 



Progress 

t 



Indicators of Success 

An increase in the state 
share of support for 
public education 



A decrease in the 
disparity among districts 
of elemeniary and second- 
ary staffing ratios 

A decrease in per pupil 
expenditure disparity among 
school districts 

A decrease in the disparity 
among districts in starting 
and mid-career salaries 



A decrease in the disparity 
among the state s subgroups 
of students (race/ethnicity, 
sex, school district, parental 
income and similar sub- 
groups) in participation m 
educational progiamsand 
educational outcomes 



Source: Meeting the Challenge, Condition 
of Education in Connecticut, Connecticut 
State Department of Education ( 1986 ) 



Commentary 

The state share of the cost of public elementary and sec- 
ondary education was 37.7 percent in 1983-84, rose to 
39 percent in 1984-85 and is estimated to be 39.7 percent in 
1985-86. 

Between 1984-85 and 1985-86 the disparity in staffing ratios 
between the 8th and 157th ranked district rose slightly from 
39 to 40 percent. 



In 1985-86 there was a difference of 60 percent between the 
95th and 5th percentile district ranked on expenditures per 
pupil. In 1984-85 this difference was 70 percent. 

In 1985-86, there was a 77 percent disparity between the low 
starting salary of $11,497 and the high of $20,322. The dis- 
parity in mid-career salaries was 85 percent; master's 
degree maximum salaries ranged from $20,100 to $37,062. 

Several indicators are used to assess the disparity among 
the state s subgroups of students. The gaps in combined 
SAT scores between white and black and white and Puerto 
Rican students narrowed in 1985-86 to the smallest differ- 
ences ever as the black and Puerto Rican averages in- 
creased and the average ,oy white students decreased. 
Males averaged 7 points higher on the verbal SAT in 1985- 
86, one point less than the prior year. Males averaged 48 
points higher on the mathematical SAT, 3 points more than 
the 1984-85 baseline level and slightly above the average 
difference over the past ten years. In 1985, 68.8 percent of 
white compared to 53.4 percent of minorities pursued 
educational activities after high school. The 15.4 percentage 
point difference between the two groups is 0.9 percentage 
points wider than the 1984 baseline difference. The gap 
between the percentage of white and minority students 
attending four-year colleges widened in 1985 as the percent- 
age of whites increased 2.1 points to 48.9 percent and the 
percentage of minorities fell 0.3 points to 26.7 percent. The 
difference in graduation rates among whites (82.5%), blacks 
(61.7%) and Hispanics (46.2%) narrowed slightly between 
1984 and 1985, as the black graduation rate increased by 0.7 
percentage points, the Hispanic rate increased by 0.3 
percentage points, while the rate for whites increased only 
0.1 percentage points. 
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are consistent with the state goals, but districts are not required 
to report progress in meeting the goals. The system is clearly 
state-based. Of the 58 indicators, only nine are aggregated 
from student or school-level data. Seventeen are aggregated 
from district data. The indicators are presented to policy 
makers and the public in two alternating publications: in one 
year they are contained in a chapter in Meeting the Challeng e: 
The Condition of Education in Connecticut, and in the next 
year they are contained in a separate publication. Indicators of 
Success . For each indicator, visual symbols and brief text easi- 
ly convey state progress. 

The heart of the district accountability program is the 
state's criterion-referenced testing program, the Connecticut 
Mastery Tests. In addition, other indicators from reports on ex- 
penditures, staff, and pcst-graduate activities of students are 
reported. The state discourages comparisons, and the data 
provided make district-by-district comparisons difficult. While 
the state reports test scores by district, the districts issue lower- 
level reports. Scores for individuals are reported to parents, 
classroom results are reported to teachers, and school results are 
reported to principals and superintendents. 

Central to indicator system development in Connecticut 
is the issue of which policy making body will report school 
data. It is the philosophy of the state to report data on the unit 
over which it has control through resource allocation — the 
school district — and to leave school reporting to the local dis- 
trict. Other practical reasons for the state's philosophy include 
the following: 

• the lag between the time that data are gathered and 
reported by the state and when they are available for 
use at the local level; 

• the smaller the unit of analysis (e.g., school), the 
greater chance for reporting or measurement error; 
and 

• the small amount of information gained by school 
reporting since most of the schools within districts 
are homogeneous (Prowda, 1988). 

Officials do not see school reporting or "school report 
cards" as part of the landscape yet. While some leaders see 
them on the horizon, there is a fear that such reporting could 



it is the state's 
philosophy to report 
data on the unit over 
which it has control 
through resource aU 
location — the school 
district — and to leave 
school reporting to the 
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pervert the testing system and result in fierce resistance from 
local districts. 



!n South Carolina, 
"school report cards'* 
provide direct com- 
parisons of a schooVs 
test results, student and 
teacher attendance 
rates, and student 
dropout rates with 
other schools in the 
state. 



South Carolina 

As a state perennially ranked towards the bottom on 
state education indicators, South Carolina was perhaps more 
ready than other states for a comprehensive program of 
education reform. The Education Improvement Act (EIA) 
of 1984 contained 61 new or expanded initiatives designed 
to affect all types of students in all grades, all members of 
the education community, and parents and businesses as 
well. A one cent sales tax increase led to a six percent in- 
crease in the regular elementary and secondary appropria- 
tion, making the state the national leader in the percentage 
increase in state aid for public education during 1984-85 
(Peterson, 1988). In return for this investment, the state man- 
dated a comprehensive system of accountability and over- 
sight. 

In addition to state leaders who continue to monitor 
the new law through a joint subcommittee, a Division of 
Public Accountability was established within the state educa- 
tion agency to monitor and report on progress in implement- 
ing the EIA. Its annual report to the State Board of 
Education, What Is the Pennv Buving f or South rarnlina in- 
cludes education indicators in six areas: academic achieve- 
ment, services to students, services to school personnel, 
school conditions, community involvement, and public con- 
fidence. A sampling of these indicators reveals: SAT scores 
increased 29 points (largest increase in nation); students in 
remedial/compensatory programs showed average normal 
curve equivalent gains ranging from 4.42 to 7.92; the per- 
centage of students scoring **3" or more on Advanced Place- 
ment examinations increased to 51 percent; teacher salaries 
reached the regional average; there was an increase of over 
300 pe^ cent in business/education partnerships at the local 
level; and public perceptions of the quality of the education 
system were more positive than prior to tiie EIA. 

The EIA provides monetary incentives to schools 
and local districts that achieve increases in academic achieve- 
ment. ^'School report cards" provide direct comparisons of a 
school's test results, student and teacher attendance rates, 
and student dropout rates with other schools in the state. In 
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addition, for test results, comparisons are provided for schools 
with similar background characteristics. This is done by group- 
ing the state's schools into five comparison groups based on cer- 
tain context variables: percentage of free-lunch eligible 
students: average teacher's education level beyond the 
bachelor's degree; dollars per pupil collected locally above 
statutory requirements; and, for elementary schools, the percent- 
age of first-grade students meeting the state readiness standard. 
In addition to percentile rankings of test results, the report cards 
present a matched longitudinal analysis of reading and mathe- 
matics test scores for the two most recent test administrations. 
Put simply, this procedure allows the calculation of score gains 
(or losses) of the same students from one year to the next. 
These gains or losses are added up at the school level and are 
used as the basis for the School Incenti/e Reward Program. 
When a school's gains rt^ach : _rtain level, that school 
qualifies for a monetary a^vard. The gain levels for rewards are 
set for each of the five comparison groups described above, 
resulting in awards to about one-quarter of the schools in each 
group. 

The EIA also requires the State Board of Education to 
adopt minimum standards for school district performance and 
to designate annually those districts that are "seriously impaired 
in educational quality." The criteria include achievement test 
results, dropout rates, accreditation deficiencies, and student 
and teacher attendance. In districts that receive an impairment 
designation, review committees conduct on-site investigations 
to identify problems and to formulate recommendations for im- 
provement. Unsatisfactory implementation of the review 
committee's recommendations can result in the withholding of 
EIA money or the removal of the district superintendent. 



The system allows the 
calculation of score 
gains (or losses) of 
the same students 
from one year to the 
next,,Mnd are used as 
the basis for the 
School Incentive 
Reward Program. 



Unsatisfactory im- 
plementation of the 
review committee* s 
recommendations can 
result in the withhold- 
ing of EIA money or 
the removal of the dis- 
trict superintendent. 



New York 



New York reports education data at the state, district, 
and school level and is currently developing a system to report 
data at the level of the individual student. In 1987 the Gover- 
nor and state legislature amended the state education law, re- 
quiring the publication of an annual '^condition of education" 
report. An important feature of the law was that it required the 
display of data, both statev. ide and by individual district, by ra- 
cial/ethnic group and gender. Published on January 1, 1989, A 
Report to the Governor and Legislature on the Educational 
Status of the State's ^^ hools is composed of two volumes. The 
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An important feature of 
New York's law was 
that it required the dis' 
play of data, both 
statewide and by in- 
dividual district, by ra- 
cial/ethnic group and 
gender. 



first provides key indicators of educational excellence and 
equity, including data on population and enrollment charac- 
teristics, participation rates, student performance, attendance 
and high school completion, postsecondary education and 
employment, staffing, and finance. The second volume is a 
compilation of profiles for every public school district in the 
state containing information on student demographics, char- 
acteristics of teachers, expenditure data, and student perfor- 
mance. 

New York's Regents Arrion Plan called for a new set 
of education goals, which are now being phased- in. The 
plan included the Comprehensive Assessment Report, or 
CAR, which is the major state accountability reporting sys- 
tem. The state is preparing reports on the schools using both 
state and local data, including a variety of state test results, 
dropout and attendance rates, graduation rates and equivalen- 
cy diploma program transfers, and socioeconomic indices. 
Based on these reports, the state gauges the performance of 
schools and identifies those in need of improvement. Local 
superintendents must present CAR results to their local 
boards at a public meeting and must implement improve- 
ment measures related to the CAR. Schools most in need of 
improvement must develop comprehensive school improve- 
ment plans through consultations among teachers, ad- 
ministrators, other professional staff, parents, and students in 
cooperation with the state education agency. To facilitate 
comparisons between schools or districts, the state publishes 
CAR data aggregated by category, e.g., community type, 
county, and New York City. 

The new Regents School Improvement and Account- 
ability Program (which still must gain final approval by the 
Board of Regents) builds upon the Regents Action Plan's 
CAR and school improvement planning requirements. The 
new program would add the following requirements: 

• annual school reports for individual school buildings 

• expanded data, at both district and school levels, on 
achievement, pupils, staff, and building conditions 

• standards of excellence for students 

• five-year education plans establishing goals, objectives, 
and strategies for school improvement 
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• a framework for maintaining accountability at the 
state, district, and individual school levels, and 



• ucfinition of the conditions under which the state 
would intervene to effect improven:ent at the local 
level, and the measures which the state would 
employ in doing so. 

Finally, the new Student Information System (SIS), a 
comprehe nsive, computer-based system, has been implemented, 
at least in part, at a number of sites around the state, including 
New York City. The purj)ose of the system is to improve the 
monitoring, decision-making, and policy- making capacities of 
both local and state officials. It incorporates a variety of data 
on individual students, including biographical, attendance, 
academic, courses and scheduling, program participation, in- 
structional managemen% and health and medical information. 
Teachers, principals, and others can use the system to identify 
and track academic and attendance trends and to identify and 
track at-risk students. The system is being implemented 
statewide through a technology network established across the 
state's districts. 

California 

California introduced two-part (state-produced and local- 
ly-produced) school performance reports in 1985 to help assess 
the impact of the state's reform legislation, to broaden the 
criteria by which schools are measured, to allow educators to 
provide input on how schools should be measured, to increase 
public support for schools by letting schools demonstrate sue 
cess, and to reward schools for effectiveness. The state- 
produced reports provide information on trends in individual 
school performance, information on how a given school com- 
pares with schools throughout the state, and information on 
how a given school compares with schools with similar student 
bodies. Compi^rison groups allow for more equitable com- 
parisons amor n ^hools and are based on a composite index 
made up of the tollowing variables: 
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parent education and occupation; 



The local performance 
report, an a4junct to the 
statC'issued school per-- 
formance report, 
provides qualitative and 
quantitative information 
gathered by local dis- 
tricts, not ths state. 



• percentage of students with limited English 

proficiency; 

• student mobility; and 

• percentage of students receiving Aid to 

Families with Dependent Children (AFDC). 

Schools are ranked from low to high on this index. 
A school's comparison group is composed of the 10 percent 
of schools ranked immediately above it and the 10 percent 
ranked immediately below it. For schools in the top 10 per- 
cent, the comparison group is the top 20 percent of schools; 
for schools in the lowest 10 percent, the comparison group is 
the bottom 20 percent of all schools. 

Indicators presented in the performimce reports in- 
clude the following: 



academic course enrollments 
satisfaction of state university course re- 
quirements 
units required for graduation 
academic achievement test results 
drop-out rate 
attendance rate 

percentage of students taking the SAT 

and SAT scores 
percentage of students taking the ACT 

and ACT scores 
Advanced Placement (AP) test results 
university- and college-going rates 
performance of graduates attending state 

colleges and universities 
instructional time 

distributions by sex and race/ethnicity of 
academic course enrollments, test 
results, and college-going rates 

amount of homework 

student mobility 



The local performance report, an adjunct to the state- 
issued school performance report, provides qualitative and 
quantitative information gathered by local districts, not the 
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state. The state has not mandated issuance of these reports, and 
this may account for the fact that very few districts are produc- 
ing them. The report provides an opportunity to address local 
conditions that affect student performance. It also allows 
schools to highlight "local" indicators. The experience of one 
of the districts visited for this study demonstrates both the 
process for completing local performance reports and the uses 
to which they may be put. 

For the last ihree years or so this district has used a 
management information system to build a database that is used 
as an accountability tool. The information collected is based on 
the district's own goals, the effective schools literature, and 
local performance indicators suggested by the state. Data are 
gathered from sun eys of parents, students, staff, and the com- 
munity; a district faculty review; a dropoui analysis; college at- 
tendance rates; financial summaries; and state and local testing 
programs. A steering committee of school principals set up 
data collection procedures, including selecting items to use in 
surveys, reviewing drafts of their school reports, and supplying 
additional information to round out their school profiles. 

The profiles produced by the district are four-page com- 
pilations of information about each school and include 
graphics. Data from the surveys and testing programs, com- 
munity demographic information, the year in which the school 
was built, the number of classrooms and teachers, average class 
size, and information on special programs are included. Prin- 
cipals are encouraged to prepare a page of unique, supplemental 
information about their schools. 

^ hile school district officials do not think that the 
presence of school report cards will necessarily make schools 
more accountable, they feel that the reports provide useful infor- 
mation for planning and improving education programs. In this 
district, the profiles are used during the year by board of educa- 
tion members and administrators for decision making. Prin- 
cipals use the profiles to compare their schools with district 
norms to ider fy areas where improvement may be needed. 
School Site Councils review the profiles as they review their 
school plans. Finally, many principals send the school profile* 
home to each family (Larick and Enell, 1988). 
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Lessons From 
Experience 



There has been a tenden- 
cy to use budding state 
indicator systems to hold 
local school systems, 
schools, and school staff 
accountable for results 
before the system has 
the copabituy for ode- 
qnately doing so. 



The net of what we have extracted from 5;tate exj)erience is 
presented below and derives in significant part from the sys- 
tems in place in the four states highlighted above, as well as 
on the other information available to the authors about other 
state experiences. The four states have encountered — and 
frequently resolved — many of the problems all states will 
encounter. While these lessons are not universally ap- 
plicable, they do pinpoint concerns which we believe every 
state would profit from addressing: 

• There is an undersiandable but often premature drive to 
report results so as to hold local school officials account- 
able; 

• There is a reluctance at the state level to assume respon- 
sibility for the quality of the indicators system; 

• There is a tentativeness with regard to the exploration of 
critical relationships among school processes, system out- 
comes such as student performance, and background or 
contextual variables; and 

• There is slow and uneven formation of the necessary 
building blocks to support an indicator system. 

A Rush to Accountability 

There has been a tendency to use budding state in- 
dicator systems to hold local school systems, schools, and 
school staff accountable for results before the system has the 
capability for adequately doing so. The reason for this is 
very understandable. For years, various leaders have ex- 
pressed frustration at **foot-dragging" within the educational 
system on assessment and accountability. The recent round 
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of stale-level reforms with concomitant increases in resources 
has provided a strong impetus to introduce local accountability 
measures in eamest. While understandable, this application of 
indicator systems may be short-sighted, because it may in time 
detract from the usefulness of the systems. 



The goal of local accountability is an appropriate one to 
be sure. Proper incentives for positive behavior at the school 
level are a legitimate end toward which state indicator systems 
ought to be directed. How to get from where we are today to 
that end is the central challenge. 

Often, critical intermediate steps have not been taken. 
Because attention and energy are directed straightaway at 
reporting local educational comparisons, the real promise of in- 
dicator systems is being overlooked. Policy makers are losing 
opportunities to understand what is happening and to analyze 
the reasons in terms of the whole state educational system so 
that clearer, more coherent directions might be identified and 
pursued. 

Indicator systems must be phased in properly, and this 
is more than just chronological ordering. State education sys- 
tems are enormously complex. They have markeu regional and 
local differences, and they are multiply governed at both the 
state and local levels. This requires that much care should be 
taken to identify and order the steps by which indicator systems 
are built at the state level. A logical progression of steps 
would first establish a statewide nronitoring capability for set- 
ting direction and sending signals, followed by a more gradual 
progression toward the use of indicators for local accountability. 

The states visited for this study, along with a few others, 
provide examples of indicator systems that are used for both 
state and local policy development and accountability. South 
Carolina, although best known for its use of indicators for local 
school accountability, has also endeavored to amass and display 
a wide range of statewide results for review by policy makers 
and others responsible for setting direction for the state's 
schools. New York State has relied heavily on reporting of 
local school results (CAR) but is also putting more emphasis on 
statewide indicators at the urging of the Governor's office miu 
the legislature. 



Policy makers are 
losing opportunities 
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As state indicator syS' 
terns have developed 
there has been too lit- 
tle cognizance of the 
ultimate test all assess' 
ment efforts must 
meet: Is what is being 
assessed a faithful 
rendering of what is 
being sought in the 
system? 



California and South Carolina have set the pace 
among the states in the use of indicators for local school ac- 
countability, through the issuance of individual school per- 
formance reports. For high performing schools, California 
extends the normal three-year review cycle and publicly 
recognizes those showing the greatest gains in achievement 
test results. In addition to publishing lists of the lowest per- 
forming schools, the state provides some technical assistance 
(OERI, 1988). South Carolina uses gains in tested student 
achievement to make monetary awards to schools and has es- 
tablished minimum criteria by which districts can be desig- 
nated as "impaired." Other states, such as Illinois, issue 
school report cards. Yet, others, like Nevada, issue school 
district report cards. Neither go as far as establishing targets 
or attaching particular consequences to results. Connecticut, 
by contrast, has adopted a different posture, focussing atten- 
tion at the outset principally on statewide results and shying 
away from local "report carding." 

In sum, the progress made by states taking the lead in 
indicator systems development should help tlieir state 
leaders set better direction for the schools. But there are still 
too few examples of state leaders acknowledging their own 
accountability for setting statewide direction as a precondi- 
tion for holding others in the system accountable. 

Appropriateness and Quality 

States have been reluctant to assume responsibility 
for making sure there is a good match between the indicator 
system and the articulated goals for change. Doing so is a 
fundamental requisite. The validity of the whole system is at 
stake. As state indicator systems have developed there has 
been too little cognizance of the ultimate test all assessment 
efforts must meet: Is what is being assessed a faithful render- 
ing of what is being sought in the system? 

The net result has been a de facto delegation of con- 
cern about appropriateness from states to localities or to 
citizens at large — chiefly through the organs of the print 
and electronic media. What is often at work may be encapsu- 
lated as follows: "We at the state level cannot deal with the 
challenges presented by matching measurement systems 
with system purposes, so we will let you — local district or 
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local citizen — do the best you can at wrestling with the im- 
plications of the reports we have issued and the statistics there- 
in." 



Our concern goes beyond the technical aspects of 
designing instruments that faithfully assess w bat people are 
trying to achieve in schools. We are also concerned about the 
political process, the synthesizing of various points of view 
toward school achievement, as a basis for selecting questions 
and appropriate instruments. Only after the political process 
has crystallized key questions can concern shift to the develop- 
ment of appropriate instrumentation. 

In South Carolina there has been a conscious attempt to 
involve, on a continuous basis, the major constituencies in set- 
ting goals for the educational system as a condition of assessing 
it. In Missouri, a similar process is just getting underway. In 
most other states, while there have undoubtedly been mission- 
determining exercises involving a wide array of interested par- 
ties, these have not been seen as integrally related to the 
development of state indicator systems. 

While there are shortcomings in matching measures 
with goals, there are many examples of efforts to improve the 
measures themselves. For example, staff of the Califomia state 
education agency in concert with the National Center for Educa- 
tion Statistics and the Council of Chief State School Officers 
are attempting to broaden the scope of dropout data collection 
to include grades below high school. South Carolina is en- 
deavoring to increase the quality of its student and teacher atten- 
dance data. Perhaps more important. South Carolina is address- 
ing a fundamental measurement problem by using longitudinal 
rather than cross-sectional data to assess school improvement. 

Tentativeness in Exploring Key Relationships 

While state leaders are gathering an increai>ing amount 
of data about the outcomes of schooliiig, school processes, and 
key background characteristics, they have a long way to go in 
the analysis of relationships among variables in these three 
areas. We have defined indicator systems as including an ex- 
amination of relationships among these three groups of vari- 
ables, and therefore, from our perspective, fully functioning sys- 
tems do not exist until this happens. 



While state leaders 
are gathering an in • 
creasing amount of 
data about the out' 
comes of schooling, 
school processseSf 
and key background 
characteristics, they 
have a long way to go 
in the analysis of 
relationships among 
variables in these 
three areas. 
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Focussing too nat' 
rowly on reporting 
of outcomes could 
render indicator sys- 
tems less, rather 
than more, useful 
as time passes. 



If there were to be a fall off, even a gradual one, 
during the next few years in student performance gains on 
tests and other outcomes, how valuable will the yield of 
state-level indicator systems be unless these relationships be- 
come the focus of attention? V/hat will the systems have left 
♦o say to state leaders and citizens in need of evidence on 
how to fine tune policies and programs, or whether to 
change direction altogether? Focussing too narrowly on 
reporting of outcomes could render indicator systems less, 
rather than more, useful as time passes. 

While it might be argued that such a fall off in test 
score gains and completion rate increases will not occur, 
there are good reasons to believe they might. After the im- 
pact of teaching to the test, Hawthorne effects, and other ad- 
justments in local behavior to make results look good have 
faded, there may be little left to report except marginal in- 
creases or decreases in outcomes. At this point the rationale 
for state indicator systems may be undermined if they only 
track changes in test scores. The need to encompass .inalysis 
is well documented in a paper prepared by staff of the New 
York State Education Department on assessing educational 
progress. We quote: 

' Any analysis must also recognize the 
process by which inputs are managed or 
manipulated to produce outcomes, positive 
or negative. In other words, the proce 
dures, schedules, layouts, and management 
processes must be examined in terms of the 
methods by which inputs are consolidated 
and employed within the school. How are 
inputs combined and implemented to 
produce effective teachers, school buildings, 
districts, and intellectually and socially 
prepared graduates, compared to the less ef- 
fective or outright failures? That is, what 
are the characteristics of the education 
process which engender increased student 
performance? 



Attention to process is important from two 
perspectives. First is the distinction be- 
tween what evaluation experts characterize 
as theory failure versus implementation 
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failure. That is, if an intetyention fails to produce 
a specific outcome according to our theory, it may 
be a result of either a failure of that theory or how 
it was implemented. The only means for determin- 
ing which is to keep a careful accounting of the im- 
plementation process. If the policy implementation 
was done as prescribed then theory failure is the 
culprit. This could indicate that the factors initially 
presumed tn have caused the problem and which 
formed t!ie basis of an original plan of action may 
have been incorrect. On the other hand, if the im- 
plementation varied from prescription then it is not 
necessarily a problem of theory failure but of im- 
plementation. In essence, being able to track 
process is critical in testing new policy interven- 
tions' (MacKinnon, 1988), 

The challenge then is to identify what it takes to en- 
courage deliberative policy making by states through the ap- 
plication of systems that not only render outcomes, but also 
provoke analysis of them. Investigations of processes that 
might generate certain effects define the essence of indicator 
systems. If areas for in-depth analysis are not identified, the 
usefulness of the systems is severely curtailed. 

While no state that we have examined has a complete in- 
dicator system in the sense just discussed, a few appear to have 
made significant inroads. The four front-running states that we 
visited — California, Connecticut, New York, and South 
Carolina — essentially have the necessary structures in place to 
begin to do such analyses. That is, they all collect data on in- 
puts, processes, and outcomes. South Carolina reports on a 
statewide basis school-level outcomes and simultaneously 
provides a fair amount of data on school processes and back- 
ground variables. California also collects and reports a large 
amount of data, but the state education agency itself does rela- 
tively little analysis in search of potential explanations. It tends 
to rely instead on Policy Analysis for California Education 
(PACE), an outside university-based group, to perform this 
function. Both South Carolina and California, however, have 
tried in their school performance reporting systems to account 
for contextual variables by "banding" schools into comparison 
categories using traditional socioeconomic and education 
measures; this in itself is a recognition of the importance of con- 
text variables. 



The challenge then is to 
identify what it takes to 
encourage deliberative 
policy making by states 
through the applu:ation 
of systems that not only 
render outcomes^ but 
also provoke analysis of 
them. Investigations of 
processes that might 
generate certain effects 
define the essence of in- 
dicator systems. 
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What seems to be misS' 
ing at his point is the 
impetus to go further. 
Why? Aside from the 
research community 
and many state educa- 
tion agency staff, 
there is little interest 
in such relational 
frameworks: state 
legislators, the busi- 
ness community, and 
the public are inter- 
ested in results . 



Although Connecticut docs not encourage the com- 
parison of district test scores, community types are identified 
for each district. Districts are classified as large, fringe, or 
medium cities; suburban and emerging suburban; or rural. 
In the near future, state policy makers anticipate the develop- 
ment of a district classification system that will permit fair 
comparisons among districts. Connecticut is working on 
categorizing communities by family background charac- 
teristics from the census. If successful, districts with pupils 
of similar backgrounds will be classified together, permitting 
fairer comparisons of educational results (Prowdi, 1988). 

New York's Comprehensive Assessment Report pro- 
gram publishes ''reference group data" or data aggregated ac- 
cording to various school classifications. These include 
public, nonpublic; type of community or district; county; 
and statewide, with and without New York City. In addi- 
tion, the state has done some work in developing a methodol- 
ogy to identify distressed areas having the most severe 
combination of educational, social, and economic need in 
the state (New York State Education Department, 1987). 
While the resulting process will be used to target educational 
resources supporting the delivery of school-based services to 
identified areas, this type of methodology could also be 
relevant to efforts to cluster schools into homogeneous units 
that could form the basis for comparison groups. 

What seems to be missing at this point is the impetus 
to go further. Why? Many of the current systems are tied to 
basic accountability and program implementation issues. 
State policy makers want to know if and how districts are im- 
plementing programs and what their effects are. Indicator 
systems, as we have described them, seem to be viewed as 
''too elegant." Aside from the research community and many 
state education agency staff, there is little interest in such 
relational frameworks: state legislators, the business com- 
munity, and the public are interested in results . There is also 
concern among some state policy makers that the knowledge 
base of what works in education is insufficient to support 
such an approach to an indicator system. Finally, state 
education agencies are busy putting out other fires and their 
research and analysis capacities are not what they once were, 
with a few exceptions. 
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Block by Block 



While many states arc increasing their data gathering 
and reporting efforts, most seem to be in the beginning stages 
of assembling the building blocks necessary for the formation 
and nurturance of complete indicator systems. There are cer- 
tain capacities, organizational norms, and systems features that 
make for a firm foundation of a successful system. 

Critical capacities include, first and foremost, a database 
that is comprehensive and well integrated. Well-organized data 
collection procedures are a must. Since the data system con- 
tains information from multiple sources, its usefulness depends 
on the degree to which data files are integrated, i.e., folded into 
a common framework so that movement from one file to 
another is possible with relative ease. The database is the rock 
on which the system is built. The sophistication of database 
development accomplished by California, Connecticut, New 
York, and South Carolina is worthy of note. At least in these 
four states there is a database capable of supporting a state in- 
dicator system. 

The state education agency must also have an analytic 
capability, however modest. The staff must be able to create 
predictive models, conduct special studies, and search for 
relationships that might provide some evidence to link what is 
happening in the educational system to possible explanations. 
New York and Connecticut stand out as having advanced to the 
point of being able to support a small number of analysts within 
the state education agency to do creative work. 

Next, the agency should consolidate, or at a minimum, 
coordinate firmly, agency-wide data gathering, analysis, and 
reporting functions. There is, we believe, wisdom in combin- 
ing all existing state education agency organizational capacities 
that deal with data gathering, analysis, and reporting. This in- 
cludes vocational and special education as well as teacher cer- 
tification and bilingual education, if appropriate. Difficult as it 
might be to accomplish this feat, it is probably the only way 
that hard pressed slate education agencies can assemble a criti- 
cal mass of resources for database and indicator systems 
development. Normally there are significant disadvantages to 
what may be called "empire building" in bureaucratic organiza- 
tions. But in this case where considerably undernourished func- 



Critical capacities in- 
clude, first and 
foremost, a database 
that is comprehensive 
and well integrated. 



The state educa- 
tion agency must 
also have an 
analytic capability, 
however modest. 



The managers of 
indicator systems 
should be afforded 
a certain degree of 
autonomy. 
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tions are involved, collecting resources together may yield 
positive rather than harmful results. 



In addition to these basic capacities, the managers of 
indicator systems should be afforded a certain degree of 
autonomy. Obviously, in a public agency this independence 
must be limited. But it is important, after the "governors" of 
the educational system have engaged in deliberations to help 
set direction for the indicators system, that its staff be al- 
lowed room to operate and be provided a buffer zone to 
produce. Connecticut and South Carolina, in different ways, 
have tackled the problem of the quasi-autonomy necessary 
for the indicators unit — South Carolina through the crea- 
tion of a partially autonomous Division of Public Account- 
ability within the state education agency, Connecticut 
through the recognition by top leadership in the state educa- 
tion agency of the need for a quasi-independent research and 
evaluation unit. 

Beyond the organizational capacities that state educa- 
tion agencies need to support an indicator system are several 
design features that can add immeasurably to the system's 
proper functioning over time. At the top of the list are three: 



Whether a state wants to report data by school or not, 
it should collect it by school and analyze it by school. The 
school is the recognized unit for the provision of educational 
services. Missouri, for example, does not intend to issue 
school performance reports at this juncture. It recognizes, 
nonetheless, that to learn important things about the quality 
of education in the state, school level data should be 
gathered. Only California, New York, South Carolina, and a 
handful of other states are able to generate extensive school- 
level data. We recommend that others join them whether or 
not they intend to issue school "report cards'*. 

Similarly, state education agencies should build up 
their staff data files, focusing on teacher data. Significant 
concern exists in almost all states about the character of the 
teaching force and the quality of school instruction. Worthy 



Whether a state 
wants to report 
data by school or 
not, U should col- 
lect it by school 
and analyze it by 
school. 



• the availability of school level data; 

• the availability of information on the 



quantity and capability of school staff; and 



• careful consideration of the "unit" of 



data collection, analysis, and reporting. 
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indicators of staff qualifications and teaching quality are few 
and far between, however. With more indicator information, 
states would be equipped to deal with teacher shortages in 
specific subject areas or geographic regions and with the chal- 
lenge of assisting districts to improve hiring and evaluation 
practices, (For a ^^prehensive discussion of the data needed 
for evaluating the effects of current and proposed teacher 
policies, on both state and local levels, see Dailing-Hammond, 
et al, 1986.) Connecticut, more so than its counterparts, has 
sought to establish a fairly comprehensive teacher database 
from which it might be able to judge whether or not the state's 
extensive teacher policy reforms are paying dividends. 

A third system feature has to do with data aggregation 
and disaggregation. Consensus on the appropriate unit of 
analysis should be achieved for all three functions — collec- 
tion, analysis, and reporting. In an ideal sense, the most ver- 
satile database to support an indicator system would be one 
where the basic units of reporting are students and teachers. 
From there one could aggregate up to class, school program, 
school, district, sub-state area, and state as a whole. Interesting- 
ly, several states, including New York and Minnesota, have 
aspirations to build such databases. Their task is formidable. 
Thirty data elements multiplied by three million students is 90 
million datum. With student mobility figures that can range up 
to 25 percent and fluctuating student characteristics, maintain- 
ing accurate files would be quite a challenge. On balance, 
some scope and range might be traded off for the capability to 
disaggregate data to the level of the school and individual 
teacher. Roughly three quarters of the states would have a con- 
siderable distance to go just to attain this level of disaggrega- 
tion. 

Necessary Steps 

For the survival and usefulness of indicator systems, 
steps not yet taken need to be taken soon. We are not suggest- 
ing that states that have moved to develop systems interrupt 
their momentum. Rather, as they proceed with systems 
development, we suggest that they close existing gaps in the 
foundations of their systems. 

We recommend these steps: 

First, engage a design process that includes all ap- 
propriate educational, business, and political lea ders with a 
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Establish specific pro- 
cedures at the state 
level to assure greater 
appropriateness and 
quality in the assess- 
ment instruments 
used. 



$teke in the health and welfare of the state's education svs- 
i£m. The explicit aim is to determine the central questions 
that a state indicator system has to answer. The broad-based 
involvement in the development of South Carolina's educa- 
tion reform program, out of which the state's indicator sys- 
tem was bom, is an example of such a conscious design 
strategy. When the state's former governor went to the 
people in 1983 and announced a program of major reform, 
he began with the recognition that, "we can raise our public 
schools from the bottom of the list, but it will take the invol- 
vement and investment of the whole state to do it." This 
grassroots effort was coordinated out of the Governor's of- 
fice and the State Superintendent's office and was actively 
supported by all of the major education associations, assisted 
by two blue-ribbon committees of 61 state leaders from busi- 
ness, education, and the legislature. Activities included local 
discussion groups, lobbying teams, speaker bureaus, school 
visitation teams, and telephone banks (Peterson, 1988), 

Second, establis h specific procedures at the 5;tflte 
level to assure greater appropriateness and gnalitv in the 
assessment instruments med 

• Consciouslv assess the im pact of current testing 
programs on instruction South Carolina has the begin- 
nings of a vehicle for addressing this issue through peri- 
odic surveys of teachers. Missouri intends to use 
surveys to ask its teachers directly what the impact of 
testing has been on instruction. 

• Pilot more hroad-gauged student fl^^ f ssments using 
small st£i^e samples The Connecticut Assessment of 
Educational Progress is a good example. In addition, 
both California and Connecticut have done considerable 
work on the assessment of higher order skills. Vermont, 
one of the few states in the nation without a state testing 
program, is proposing a pioneering effort to assess stu- 
dents on the basis of work portfolios. 

• Build checks and balanc es into the system bv developing 
a core indicator syst em complemented by surveys of at- 
titudes and sample school probes The core system 
should be composed of basic indicators of important in- 
puts, processes, and outcomes collected over time to cap- 
ture aggregate effects of educational policies. This basic 
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model, of course, would vary from state to state, 
depending on a host of factors such as state and 
local capacity, policy focus, priorities, and reform 
activities. The system would be an "open" one, 
easily modified to include new research, changes in 
policy focus, feedback resulting fiom the system, 
and so forth. This core system should be aug- 
mented with special surveys and studies of a 
selected sample of schools ("weathervane 
schools") to collect information on the attitudes and 
opinions of various groups and to provide an in- 
depth, qualitative asssessment of program impacts. 
The results of these local "probes" can also be as- 
sayed against th'^ yield of the core system to pro- 
vide a chcvk on the validity of the indicator system. 

Third, focus statewide data collection on selected vari- 
ables identified in the st aters variant of the framework sug- 
gested in Figure 2 . Such variables as amount of active class- 
room teaching time, amount of class time teachers use to 
question students, percentaj^ t of part-time and substitute 
teachers employed, extent of curriculum tracking, and amount 
of released time teachers have to observe other teachers have 
been mentioned as deserving o' examination for inclusion in an 
indicators system where exploring relationships is as important 
as measuring outcomes. (For a useful discussion of these vari- 
ables, see Darling-Hammond, et al, 1986.) Each variable must 
be scrutinized to see if a relationship to performance has been 
established by research and can be adequately measured in an 
indicator system. By incorporating a greater capacity for policy 
anaiys^*s into the system, states will be in a position to be more 
helpful to localities seeking to improve schools. While it may 
be difficult to prescribe proper remedies, such complete sys- 
tems should provide clues that would be missed in haph izardly 
constructed systems. 

Fourth, emphasi ze school-level data gathering and 
analysis as a minimum requirement and gather data by student 
if possible . The purpose here is not to increase the possibilities 
for shock value in public reporting of data by school or by stu- 
dent descriptors such as race/ethnicity or socioeconomic 
status, but to afford state-level policy makers the perspective 
for more refined judgments and to provide them with greater in- 
sight into school processes and school and community context. 
The ideal would be more analysis and less reporting of data by 
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school and student levels until the indicator system can be- 
come firmly establisned. This would not only make for a 
healthier indicator system but a healthier education system 
as well. New York's work on student database development 
and the work on indices of community need discussed ear- 
r are illustrative here. 

Finally, build the necessarv i nfrastructure to supp ort 
the developni ent of indicator systems by allowing for the par- 
tial autonomy of the analytic unit w orking on the data base 
and building in prnnpr protections hefnre data are collected 
to preserve the ability of the system to f ulfill the baseline 
purposes intended for it by state leaders . Examples of 
promising practices in several states have been mentioned 
above. 
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